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Abstract 

In this paper we study controlled continuous time random walks (CTRWs) and 
heuristically derive pay-off function dynamic programming (DP) equations which turn 
in the limit of standard scaling to fractional Hamilton Jacobi Bellman (fHJB) type 
equations. This paper aims to extend the results from !li in a controlled setting. 
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Introduction 



^.1 Plan 

■)—> 

^We introduce the controlled process and write the DP equation for the pay-off function. 

i_^/e scale the waiting times and the jump sizes by a parameter r raised to specified powers, 

and obtain the limiting equation for the optimal pay-off function as r — )■ 0. Then we make 

(flight modifications to the process, such as including waiting and jumping reward functions 
(■r^s well as an inner motion during waiting, and write corresponding DP equations for these 
^^jphy-oS functions, again looking at what happens as r ^- 0. After this we study a generalised 
vQversion of such a process and arrive at a general limiting equation for the pay-off function. 

rn 

01.2 Notation 

^-H _ • a+Dx left-sided Riemann-Liouville fractional derivative of order /3 G (0,1), with a 
^ ' lower integration limit a, which can be — oo 

/\ • h-Dx right-sided Riemann-Liouville fractional derivative of order (5 € (0,1), with a 
C^ , lower integration limit 5, which can be cxd 

• a+Dx left-sided Caputo fractional derivative of order /3 G (0, 1), with a lower integra- 
tion limit a, which can be — cxo 

• b-Dx right-sided Caputo fractional derivative of order /3 G (0, 1), with a lower inte- 
gration limit b, which can be oo 

• -r-p the generator form of the fractional derivative of order /3 G (0,1) 

• [x] is the integer part of x G M 
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• For a complete metric space S, Coo{S) consists of functions / such that hma,_>.o f{x) = 
0, i.e. Ve > 0, there exists a compact set K such that sup^^^ 1/(3^)1 < e 

• 5(M'^) = {/ € C7°°(M'^ : VA:,/ G N,\x\^V^f{x) € C7oo(M'^))} is the Schwartz space of 
rapidly decreasing functions 

• C'^(M'^) is the Banach space of k times continuously differentiable functions with 
bounded derivatives on M , the norm is the sum of the sup-norms of the function 
and all of its derivatives of orders up to k inclusively 

• Isz^A} stands for the indicator function of a set A, which is equal to 1 when its 
argument z is in the set A, otherwise it is 

2 Fractional calculus 

There are numerous articles and books with an introduction to fractional calculus and its 
uses, see e.g. [5, J5] and [6]. Let's recall the main definitions and links for convenience of 
the reader. The Caputo derivative is one of the several versions of the fractional derivative. 
It is closely interlinked with the generator form of the fractional derivative and the Riemann 
Liouville derivative in the way described below. 

The left-sided Caputo derivative for f3 E (0, 1) is defined for / smooth on [a, oo) by 

The right-sided Caputo derivative for /3 G (0, 1) is defined for / smooth on (— oo, 6] by 

The left-sided Riemann-Liouville fractional derivative for (3 G (0, 1) is defined for / 
smooth on [a, oo) as follows: 



I d_ 



a+D^f{x) = —^ -— I f{y){x - y)-^dy. (3) 



The right-sided Riemann-Liouville fractional derivative for (3 € (0, 1) is defined for / 
smooth on (— oo, b] as follows: 

The left-sided fractional derivatives operate on a < y < x, whilst the right-sided frac- 
tional derivatives operate on x < y < b. 

For every function / which is everywhere continuously differentiable [6], 

a+D^fix) = a+Dff{x) + Y^^/(«)- (5) 

To prove this do a change of variables x — y = z and differentiate the integral, then let 
z = X — y to get the result: 



a+D^J{x 



1 



d 



r(i - /3) dx A 

1 d '■0 



r(l - /3) dx 

1 d 



x—a 
x—a 



1 



r(i-/3) 



r(i - /3) dx 7o 

/3j-^„ „ , \ , f^'"" df{x - z 



(x — a) '^^/(x — X + a) + 
1 



dx 



f{y){x-y) '^dy 
f{x — z)z~^dz 
f{x — z)z~^dz 
z'l^dz 



r(i-/3) L 

Q.E.D. Similarly one can analogously prove that 



{x-a)-^f{a)+ r^ix-y)-^dy 



dy 



-D^xfi^ 



Dff{x) + 



(b-x) 



-P 



-fib). 



(6) 



(7) 



r(i-/3) 

The equations ([5]) and ([7]) motivate the definition of the it regularized Caputo derivative. 
When / is not differentiable the regularized Caputo derivative is defined as 



a+ 



dffi3 



d 



r(i - /?) dx 



f{x - y)y ^dy 



fia 



(x — a) ^ 

r(i-/3)" 



(8) 



The generator form of the fractional differential operator for every / at least from the 
Schwartz space 5(M) is defined as follows: 



-i^ = n^)L (^(^-^)-^^^))^w^ 



/?G(0,1). 



(9) 



This also applies to all / which are bounded and 7-Holder continuous, for 7 < /3 [2j. We 
can also let / be such that /(x) = for x < a and C^[a,oo). The reason for which this 
fractional derivative is referred to as the generator form one, is that for a /3-stable Levy 
motion the generator is exactly, 



A,f{x) = -f^ [Uix + y)- fix))^ 



(10) 



which knots fractional calculus with probability. 
The dual operator to Ajj is defined as follows: 



Alfix) 



r(-/3) 70 



(fix -y)- fix)) 



dy 



y 



1+/3- 



In particular 





r(i-/3) 


-A}l{t>o}, 


here the dual to A operator 


A} = a+Df = 


- D^ 

- a+^x 


ith a = —00. 







(11) 

(12) 
(13) 



For every a < 0, if f{x) = for x < 0, then the generator form coincides with the 
Riemann-Liouville fractional derivative and with the Caputo fractional derivative. In case 
a = 0, f{x) = for X < 0, the generator form is equivalent to Caputo form, but not to the 
Riemann-Liouville form [3]. To prove the generator form is equivalent to the Caputo form 
for a < 0, and f{x) = if a; < 0, integration by parts yields [3] 

1 



[/(^_y)_/(^)]___y-/3-ldy 

y-Hfix-y)-fix)) 



-/3r(-/3) 



oo 




1 /"OO J 

(15) 
For / bounded and continuously differentiable, for y — )• 

y-^[fix-y)-f{x)] = Oiy'-''), (16) 

and for y ^ oo 

y-P[f{x-y)-f{x)] = 0{y-^). (17) 

Hence the boundary term in (J14p is and 

1 l"^ d 

[f{x -y)- f{x)\y-^dy 

f'{x - y)y~'^dy 



/3r{-P)Jo dy 
1 



r(i-/?)yo 



1 r dfiy), ,^fs, 
(x-y) Pdy 



r(l-/3)y-oo dy 

= {-oo)+Dlf{x) 
= a+D^J{x), Va<0. (18) 

(19) 

3 A continuous time random walk (CTRW) 

For convenience we recall here the main definition of a CTRW. 

Definition 1. For i € N, waiting times 7^ G M^., and jump sizes ^i G W^ we let (71,^1), 

(72)^2)7 • • • be a sequence of i.i.d. pairs of random variables (r.v. 's) such that the distribution 

of "fi is given by the probability measure v[dr), and the distribution of jump sizes is given 

by the probability measure f^{d^). Let X{n) = Y17=i'yi- Define the inverse process Zx{t) = 

inf„{n : X{n) > t}, which is continuous in time. A continuous time random walk (CTRW) 

with jumps ^i S M and waiting times 7j € M+ is defined as the process in continuous time 

given by 

Zxit) 

Y{t) = Yz^(t) = Y. Ci, (20) 

i=l 

arising from the measures v and \x. 



One interprets this as follows: the particle waits for a time 7j € M+ and makes a jump 
^i G M*^, then it waits for a time 7i+i € M+ and makes a jump ^j+i € M'^, e.t.c. for z € N. For 
now we assume that the waiting times and the jumps are not time or position-dependent 
random variables. Later in the paper, we do consider this complication too, see Chapter 8. 

In a CTRW each jump is characterised by a waiting time and a random jump length 
that are i.i.d. random variables with probability densities z^ and /i. As it is usually done in 
the theory of CTRWs, e.g. [1] and [13], let us assume that for i G N, 7j and ^i belong to 
domains of attraction of stable laws: for n — > oo, 

^^'^^^ ~ VfR \^ g ' (21) 

r\>n r(/3 - l)nP 

(3 € (0, 1), and concerning the jumps we assume for n ^- oo 

/ / 'y{i)ds{i)t^{dy) ~ 4^ / a{£,)dsi, (22) 

J\y\>nJn n"' Jn 

where a S (0, 2], o" is a spectral measure on the sphere S'^^^, and for every Borel Q G S'^^^ 
which can be any sector of the unit sphere S'^~^ . If a = 2 we instead assume that 



/ 



y^li{dy) < oo. (23) 



Remark 1. Note that if 'yi,i E N were exponentially distributed, i.e. f3 = 1, then this is 
a standard random walk and the DP equation for the optimal pay-off will be the Hamilton- 
Jacobi- Bellman (HJB) equation with a classical time derivative d/dt. 

Remark 2. As it is well known, see J^', Q/, the distributions of normalized sums of i.i.d. 
copies of such r.v.'s converge to a stable law. 

4 Fractional Hamilton Jacobi Bellman (HJB) for a controlled 
scaled CTRW 

If we pay attention to the times when the jumps take place, then the process Y{t) has 
Markovian properties at the jump times, which are the renewal times for the process studied 
only at jump times. It is possible that as the particle waits or jumps, it has to pay a price 
for doing so, or it receives reward. Here we will assume that we can control the distribution 
of jumps so that we can maximise our reward. We highlight that the jump control may be of 
any nature, and we denote by U the set of all possible controls at any particular jump, the 
set of controls being the same for every jump. By U denote the set of all possible controls 
at all the jump times: 

U = {u = {ui,U2,...)},Ui GC/,i eN. (24) 

Control brings about dependence of the probability measure /i on n E [/. When control is 
introduced into the system, for i G N, ^j become dependent on control, so we write ^i{ui), 
and the latter are control dependent and not identically distributed random variables. Their 
distribution now depends on control, hence the notation fiuid(,) appearing later. Now 

Zxit) 
Y{t,u)= Y^ C.iui) (25) 



is our controlled CTRW. We are not going to control the distribution of waiting times, hence 
Zx{t) is the same as before. 

Assume that the jumps are costless and pay-off is only the terminal one given by the 
function So{y), and that there are no rewards after the process has terminated. Given Soly), 
the optimal pay-off function S{t,y), is then defined as follows 



S{t,y) = snpESo{y + Y{t,u)), t > 0. 



(26) 



This represents the optimal reward the particle receives when the time until the end of 
the process is t and the particle is in position y. Unless specified otherwise, we will refer to 
the time until the termination of the process as t. 

For convenience let us denote the waiting time until the next jump by the random 
variable 7 with distribution u{dr) just as the previous waiting times 7^, i G N, the waiting 
time until a jump ^ with distribution //^(d^) occurs after time t. Then 



/oo 
u{dr 



(27) 



is the probability that there is no jump within time t to the end of the process. If a jump S, 
takes place at a time r £ [0, t], then the expectation of the pay-off 5 at {t — r,y + $,) will be 
represented by the integral J^d S{t — r,y + ^)/iu((i^), where 0<r<t, r = t — 7. 
Consequently, S{t, y) satisfies the following DP equation: 



S{t, y) = sup 



5o(y)P[7 >t]+ I ES{t 
Jo 



r,y + ^{u)\^ = r)u{dr) 



sup 

ueu 



00 r^ r 

Soiy) I u{dr)+ / S{t-r,y + i)^i^{di)u{dr) 
t Jo Jm.'^ 



(28) 



The first term represents the expectation of the pay-off if a jump has not occurred 
within time t to the end of the process, and the second term represents the expectation of 
the pay-off if a jump has occurred within that time. 

Now let us make the following scaling: 7j i-^ 'y^r'^^^, and £,i 1-^ CiT^''^, i G N, where 
T > and is small, also /3 € (0, 1) and a € (0, 2]. For this scaled process, we will denote the 
optimal pay-off at time t and position y by S'^{t,y): 



S-{t,y) = suvES^{y + Y{t,u)). 
ueu 



(29) 



Our previous S{t,y) is S^(t,y). We are interested in the limit r — ?> 0. The DP equation for 
S'^{t, y) becomes 



S^{t,y) = sup 



sup 



S^{y)Ph>t]+ / ES^t-r,y + au))h = r)u{dr/T'/^) 



r/T^/^) 



/"CO rt f 

S^y) / Hdr/r'/^) + / / S^t -r,y + Oliuid^/r'/'^Hd: 
Jt Jo Jr'' 



(30) 



We assume that as r ^- 0, S"^ has a limit S{t, y) which is smooth. This is the usual 
assumption of a heuristic approach to the deduction of differential equations of dynamic 
programming. We wish to identify the limiting equation satisfied by S{t,y). 



By adding and subtracting identical terms, we can bring equation (|30p to the following 
form: 



sup 



S^ {t,y)^ sup (s^y) r v{dr/T^/^)+ f f S^ (t ~ r,y + ^)f^4dar'/"Mdr/r'/^) 
ueu \ Jt Jo Jr^ 

POO rtr '^ r . 

= SUy) u{dr) + sup / (s^{t-rT'^^,y + ^T'/-)~S^{t^rr'^^,y) + S^t~rT'^^,y))^,^{dOHdr) 

Soiy) r '^(^'■) + ^ r I - (^'(* - ^^'''^ y + ?^'^") -S^ii^ ^^'^^ y)) f^nidOuidr) 

JtT-i/f Jo JR'i ''" ^ ' 

+r / -[S^{t-rr'^^,y)-S^{t,y)]^^4d0u{dr)+ / Sli,2/)M4dC)Kd^ 

Jo Jm-^ '^ Jo Jwi'i 



+ sup 

ueu 



Soiy) I u{dr) + r 

tr-l//3 ^0 



Rt* . 



+T 



Jo Jm'' 



S-{t-rT''^,y)~S-{t,y) 



liu(d^)v(dr) + 



fiu{d£,)i^{dr) 



S^{t,y)^iumv{dr) 



Note that since S'^{t,y) is independent of S,, integration with respect to ^ gives us 1 in the 
inner integral because Hu is a probability measure, and also since there is no dependence on 
control in the integral, the term can be taken out of the supremum. 

We divide the whole equation by r and proceed as follows with the last term of the RHS: 



,tT-l//3 



Jo Jr' 



S-{t,y)iiu{di)u{dr) 



S^{t,y)u{dr) 



S^{t,y)iiu{di)v{dr) 
S^{t,y)fi^{dOiy(.dr) 
S^{t,yMdr). 



(32) 



We shall see both of these integral terms cancelling out with other terms in the equation. 
The term 5'^(t, y) on LHS cancels with sup^gjy j^ S'^it, y)v{dr) on the RHS, because z^ is a 
probability measure and the latter integral is independent of control U . 
Now we work with the second to last RHS term: 



r-tr-V/S 



/ / 



S^t-rr'/'',y)-S-{t,y) 



fJ'uidO^idr) 



tr-^/f 



tr-^lf^ 



S-(t-rT^IP,y)-S-{t,y) 



S-{t-rT^/^y)-S^t,y) 



S-{t-rT^l^y)-S-{t,y) 



u{dr) 

u{dr) 

u{dr), 



(33) 



and in the second integral the first term is due to the boundary condition we have chosen. 
The second term in the second integral cancels out with — ^ j^-i/p S'^(,t:y)i^{dr), whilst the 



(31) 



first integral, as r — )• gives us a neat expression presented just below. As r ^ we liave|^ 

's-it-rry^,y)-S-it,y)' 



u{dr) ^ A*pS{t,y), 



(34) 



where A*g is the dual to the generator A^ for Xt, the limiting process of X'^ as r — >■ 0, where 
Afs is defined in (jlip and represents the generator of the /3-stable Levy subordinator. 
Now, we recall from the definition of the stable-like probability measure z^: 



v{dr) 



1 



r(i - P)n^ 

as n ^ oo. In our case n = tT~^'" — t- oo as r ^ 0. 

By the means of the above condition and the essential assumption that 



(35) 



S^{t,y)^Sit,y), asT^O, 



(36) 



where S{t, y) is smooth and belongs to the domains of the limiting stable generators L'% 
and Ap, the first term of the RHS, as r ^- 0, gives us 



lim 



Sl{y) 



v{dr) 



Soiy)t- 



r^o r Jt,-i/^ ' ■ r(l-/3) 
As for the other term on the RHS, as r — )• we have ^ 



(37) 



tr-V/3 



S-'it - rT^,y + ^rV°) - S^{t - rr^l^, y) 



flumHdr) ^ L^S^t - rr^/^ ,y) ^ Lj{t,y),{38) 



where Lq, is the generator for Yt, the limiting process of V^ as r — t- 0. Note that the stable 
like generator Lq, has the following form for a > 1: 

f{y + r)-f{y)-f{y)r 



Lafiy) 



\d+a 



-uj{u)dr; 



(39) 



and the general stable symmetric law generator for a > 1 has the form in polar coordinates: 

f{y + r)-f{y)-{r,Vf{y)) 



Lafiy) = / d\r\ 

Jo Js^--^ 



|a+l 



-u}{dS), 



(40) 



where io is an arbitrary symmetric finite Borel measure on S" ^. 
Whilst for Q G (0, 1) 



Lafiy) 



fiy + r)-fiy) 



ijjiu)dr- 



(41) 



and the general stable symmetric law generator for a > 1 has the form in polar coordinates: 

fiy + r)-fiy) 



Lafiy) = / d\r\ 

Jo Js-i-^ 



|a+l 



-ojidS), 



(42) 



The above limiting procedures and that r now cancels out in front of the integral terms 
of the RHS, now yield us: 



m 



^This convergence follows from our assumption on v in (|2ip . and the details can be found for example in 



— --t ^5o(y) + sup 



LlS{t,y) +AlS{t,y). 



(43) 



Here Af^ is the generator form of the fractional derivative and A*a is its dual operator. 
Due to the link between the Riemann-Liouville and the generator form of the fractional 
derivative when a = we can write this as: 



o+D^J{t,y)-- 


= sup 


'LlS{t,y) 





(44) 



recall that the latter fractional derivative qDx is the Riemann-Liouville type. In case of no 
control this equation turns to 



^}S{t, y) = -LaS{t, y) + S'o(y)yl^l{j>o} 



(45) 



which appears in [2]. 

The equation (j44p is fractional in time and in space, and is of Hamilton Jacobi Bellman 
(HJB) type. For controlled Markov diffusion processes instead of the fractional equation we 
have the second order HJB equation jl2] : 



d_ 
di 



S{t, x) + H{t, X, DxS{t, x),DxxS{t, x)) = 0, 



(46) 



where H is some Hamiltonian. Thorough analysis of equations such as (1460 is presented 
in [12] . These HJB equations have been proven to have viscosity solutions, which form a 
particular class of weak solutions, and when we analyse our fractional HJB type equation 
it is reasonable to look for viscosity solutions too. This topic will also be discussed in the 
authors' further research. 

5 Fractional HJB for a controlled scaled CTRW with running 
rewards 



If we now take into account the reward for jumping from position y, namely f{u,y), then 
the DP equation for the optimal payoff becomes the following: 

ft 



S^{t,y) = sup 
ueu 



[5J(y)] / u{dr/T^IP)+j I [S^t-r,y + + f{u,y,0]f^u{dar'/''Hdry/'') 



.(47) 



We re-write it in the form: 



S''{t,y) = sup 



-l//3^ 



t Jo JRd 

+ f I ifiu,y,OhuidC/r'h'^idr/T'/f') 
Jo JR'* 



.(48) 



Now, we make an assumption on the function / as follows: for A > 

/(n,y,Ae) = A"/(n,y,0, (49) 

i.e. / is homogeneous of order a, where a G (0, 2] is the stability index we've met before. 
Denote 

F{u,y)= f fiu,y,C)f^u{dC). (50) 



We have for r — )• 0: 



ir-l//3 



T Jo Jr'' t' Jo Jk'' 

F{u,y) u{dr) ^ F{u,y). (51) 



Therefore, the full limiting equation with the included reward function for jumping, 
obtains the following form: 



— — 5o(?/) + sup 



LlS{t,y) + F{u,y) + AlS{t,y) 



(52) 



Note that AZS{t, y) is in independent of control, so it can be taken out of the supremum. 
If we also include waiting rewards into the system, the DP equation for S'^{t, y) then becomes 



S^{t,y) = sup 
ueu 



Sl{y) j u{dr/T"^)+ / [S^{t-T,y + i)]^iu{di/T^h^{dr/T^/P) 
t Jo Jw 

* / (/(^,y,e))^n(dC/ri/-)Kdr/ri//3) 

poo ft f 

+ t / guAyHdr/T^) + / / 9uAy)rf^u{dar'h'^{dr/T'/^) 
Jt Jo Jr-^ 



and for r — )• we have 
i'{dr/T 



9u,t{y) [\./j„i,i/0\ _ 9u,t{y) '"*^ 



and 



guAyy^^uid^/r'/n'^idr/Ty^) = -g^Ay) 

T Jo Jr'' t' jo 

A/13 



tr-V^ ^l//3^d^ 



-1+13 



uAy)^{tT~"')'-' = 9uAy)t'-'- 



(53) 



^,='-''^{tr-''')-'=9uAy)t-^ (54) 



(55) 



This gives us: 



— ^5o(y) + sup 



LlS{t, y) + 2guAy)t^~^ + Hn, y)] + AlS{t, y) 



(56) 
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6 Fractional HJB for a scaled CTRW with inner motion 

We can also add particle walking during waiting, which is an arbitrary Feller process M^"(y) 
and which we will refer to as inner motion. In [8j such a process, i.e. with inner motion, is 
referred to as a process with a semi-Markov chance interference. Our DP equation obtains 
the following form: 



S'^{t,y) = sup 



{E[S-{Wny))P{r>t)]+ / / E{S-{t-r,Wr{y)+Oh = rhu(dUr'hHdr/T'/^) 

Jo JRd 

(57) 

where W]^ is a limiting Markov process and a generator L for the limit of the scaled 

controlled CTRW, as r ^ 0. We scale the waiting times for this inner process by r^' ^ too, 

and also 



ES{Wriy)+^,t-r)= [ S{z + ^,t - r)Pr{y,dz), 



(58) 



and 



E[S^oiWr{y))]= S^,iz)PtiWriy),dz), (59) 

jR'i 
and in the limit as r ^- the inner motion has a generator B. So we have the following: 



+ / 

Jo 



Hence, 



sup 



tT'-^/P 



+ 



tr-l//3 



5^(t,y) = sup / / S^{z)Pt{y,dz)v{dr/T^I^) 
u<=U Ut Jw 

S-{z + e, t - r)Pr{y, dz)fiu{dar^hHdry/^) 

S^{z)Ptr^/p{y^dz)v{dr) 

S-{z + ^T^l'^.t - rT^/^)P^^,/, (y, dz)^i^{dOy{dr) 

S'' {z)P^^i/p{y,dz)u{dr) 

Hu{d^)iy{dr 



sup 



4, S^z + ^Ty'^,t - rT^IP)P^^,/p {y, dz) - S^y, t - rr^^) 



+ f "" (r'-l^^^^^l^^l^^l^^S^it,^ .idr) 



,(60) 



+ 



tr-l//3 



OO /"OO /• 

S^{t,y)u[dr)- / S^{z)P,^,/,{y,dz)v{dr) 



sup 



S-{t-rT^l^y)-S-{t,y) 



v{dr) 







S^{t - rr^l^, z + Cr^/") - S^{t - rT^/^, z) 



PrT^/fi(y,dz)^iu{dO^{dr) 



+ 



tr-^/l^ 



Sr(t - rT'll', z) - S^t - rr^l^.y) 



i/P 



PrT^/piy,dz)u{dr) 



■ (61) 
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The last summand can be represented in the form: 

Z"*"''^" f S^{t-rT^I^,z)-S^{t-rT^I^,y) 
Jo 



PrTi/^iy^dz)u{dr) 



S^t - TT^If^, z) - S^t - tt'^I^ y) 



Jw '^ 

S^{t - rr^/l^, z) - S^{t - rr^/P, y) 



Prr^/aiy^dz) 



PrT^/p(y,dz)v{dr). 



(62) 



'tr-l//3 jRd T 

As r — 7- 0, due to the boundary condition the two terms in the second integral are O's 
and the first integral satisfies: 

S^{t - rr^l^, z) - S^{t - rr^lf^, y) 



-P^^^/,{y,dz)u{dr)^B--S{t,y) 




As for the term 



we add and subtract 



and this gives us: 



T JtT-^/f> 



S''{z)P^^i/p{y,dz)v{dr), 



Sl{y)v{dr) 



T JtT-^/P 



1 



(63) 
(64) 
(65) 



[j^,S-{z)P^^,/,{y,dz)-Sm] 



S'^{z)P^^i/p{y,dz)L'{dr) 



tT~yi3 



u{dr) + - r Sl{y)u{dr). 



(66) 



Now, 



^^i//3-i f S^{^) - Sl{y) 



and 



, i/« Pt,i/,{y,dzMdr) ^ / tr'/f'-'B^S^iyMdr) ^ 0,(67) 



1 r°° f-P 

r JtT~^/P r(l - /3) 



(68) 



Hence the equation for 5'(t, y) with an inner motion during waiting, as r — > 0, is 



— —t ^S'o(y) + sup 



LlS{t,z) + B^S{t,y) +AlS{t,y) 



(69) 



If we also add the rewards for jumping and rewards for waiting, then the extra terms 
with gu,t{v) and F{u,y) appear, yielding: 



-— —t ''S'o(y) + sup 

J-I-L-Pj u& 



LlS{t, z) + B^S{t, y) + 2guAy)t'-^ + ^iu, y) + AlS{t, y) 



(70) 
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7 The HJB written backwards in time 

When working with time non-homogeneous processes it is more convenient to change the 
notation. Now S{t, y) will denote the optimal pay-ofF at position y and time t, where t is 
the time elapsed since the beginning of the process. Now we will have S{T, y) known, where 
T is the terminal time of the process. Now S{t, y) is defined as 



S{t,y) = supE[S{T,y + Y{T -t,u))]. 
iidU 



(71) 



We can write the DP equation backwards, which means time is moving forwards and t 
denotes time elapsed since the beginning of the process. We shall see the differences this 
brings to the DP equation as r ^- is that firstly, instead of the dual operator A* we will 
have the generator A and secondly, that the boundary term will depend on T too. 



5^(t,y) = sup S-{T,y) / u{dr) 

u€U [ J{T-t)T-i/l3 

+ T I I fiu{dt.)^{dr) 







^ ;^^"*^^"'' f S^{t + rry^,y)-S^t,y) .^^, .^ , 

AT~t)r-^/f^ . 

+ / / S^{t,y)iiu{dO<dr) 

Jo JR'' 



(72) 



We divide the whole equation by r. As r ^^ the first term on the RHS gives us the 
limiting boundary term 

(T-t)-^ ^ 



-S{T,y). 



r(l-/3) 

Note that it is not dependent on control and hence we can take it out of the supremum. 
The second term on the RHS gives us, as r ^^ 0, 



(73) 



(T-t)r-i/^ [5r(^ + ^^l/;3^y^^^l/a)_^r(^^^^l//3^y)] 



^K'' ''" 

The third term we write in the following form: 

(^-*)^"'' f [S^t + rT'/^y)-S^t,y)] 




fiidOHdr) ^ LaSit,y). (74) 



oo 



[S-{t + rT^/^,y)-S-{t,y)] 



Jk'^ 't 

^ [S^{t + rT^l^y)-S^{t,y)] 

T 



For the first integral above: 



^i.u{dC)v{dr) 
^iu{di)v{dr) 
^i.u{dC)v{dr) 




As for the term 



I^u{d0v{dr) ^ ApS{t,y). 



(75) 



(76) 
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(r-t)T-i//3 
and 



S^(t + rri//3,y) 



^u{dC)^{dr) = due to the boundary condition S''^(t, y) = for t > T, 



1 



(r-i)r-l//3 



T JO 
1 roo 



i r / S^{t,y)iiu{diy{dr)-- 

T Jo Jr'' T' J{T-t)T-^/P 



(77) 
S%t,y)iiu{dOy{dr) 

i 

S^t,y)fiu{dOHdr). (78) 



The first integral in the expression above for the last term of the RHS cancels out with 
S'^{t,y) on the LHS, -S'^{t,y) and the second term in the above representation cancels out 
with the term 

- r [ ^^^^^fiuidO^idr). (79) 

Therefore we arrive at the forward equation as r — )• 0: 



= ^^ rn'^T'^^ + ^MLlS{t, y)] + ApS{t, y). 



(80) 



If jumping and waiting rewards are present, then we obtain: 



{T-t)~PS{T,y) , 

r(i-/3) u& 



LlS{t, y) + 2gu,t{y)i^~^ + ^(^> v) + ApS{t, y) 



(81) 



If the inner motion is present too: 



{T-t)-PS{T,y) , 

r(i-/?) +S^ 



L^^(i, y) + i?"S(i, y) + 25„,t(?/)ti-^ + F{u, y)] + ^^^^(t, y) 



(82) 

Here oo-Dx = Aj^. The proof is just below and consists of changing variables x — y = z, 
differentiating under the integral and using that —/3T{—/3) = T{1 — j3) for j3 G (0, 1), and 
changing variables again: —z = y: 
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1 d r°° 

1 r] fO 

fix-z)\z\~^dz 



r(l - /3) dx 



5 






r(-/3)._oo 



oo 



1 f"" 

= -f(Z^X (/(x + y)-/(x))|y|-'5-idy 

= ^/3/(x). (83) 

The boundary term appearing after integration by parts is because \z\~^ — )• as 

\z\ — 7> OO. 

8 Position and time dependent HJB 

8.1 Position dependent HJB 

It is known that for an arbitrary Feller process Yf with a generator L" it is always possible 
to construct a family of measures fJ,u,T,n, such that 

Y;:=Y;:_,+Cn(.yn-l), (84) 

where S,n,n G N are r.v.'s with distributions given by fJ.u,T,n, so that 

/ /(^'^ + ^)-/(^'^) ^,,,(,,,0 - L-fit,y). (85) 

Consequently for n = [t/r], by theorem 19.25, attributed to Trotter, Sova, Kurtz and 
Mackevicius, and theorem 19.28 in [3], 

Y^l/r] ^ Yt. (86) 

The DP equation is 

S^it,y) = sup (sUy) r v{dr/T^'^)+ [ [ S^t - r,y + Of^u,yAdO''idr/T^^^)) ■ (87) 
ueu \ Jt Jo JR'i / 

In the limit r ^- this turns into 

= rn^ a, t~^So{y) + sup[L«^(t,y)] + A*^S{t,y). 
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If the inner motion and jumping and waiting rewards are also included, we have: 




Here the generator L" is an arbitrary generator of a Feller process Yt and A*^ is the dual 
operator for a generator ^ of a /3-stable Feller process Xj, and B is the generator of the 
Feller process which is the inner motion. 

8.2 Time dependent HJB 

We could have stable position dependence together with general time dependence. In this 
case 

X^ = X;_i+7;(X;_i). (90) 

We build families of 7^, n = [t/r] G N, each with the law ut^riXJ^-i, dr) so that we can 
have a discrete time Markov chain approximation of an arbitrary increasing Feller process 
Xt with a generator A: 



f{t + r)-fit) 



Ur{t,dr)^Af{t). 



'0 T 

Consequently by theorems 19.25 and 19.28 in [3] 

For instance, in the case when A is stable-like we have: 

at) 



u{dr) 



(91) 



(92) 



(93) 



Itr-^/m r(l - /3(t))(tr-V/3W)/3W 

as r — )• and so tr^^'^ — )• 00. Note that now the measure v depends on time t elapsed 
since the beginning of the process. Also, we include waiting time rewards. Now boundary 
condition is S{T, y) is known and we write the DP equation: 



S^{t,y) = sup 



/■OO 

[S^T,y)+guAy){T-t)] ut^dr 

JT-t 

+ I I [S^{t + r,y + i)+ 9u,t{y> + fiv + e, u)] fiuA'i^/^'h^tAdr) 
Jo Jr'' 



We assume that 



1 /■* 
lim — / rvT-idr) = Ki(t), 
^^0 T Jo 



and 



and that 



lim — 

T^o r 



l^ridr) = K2{t), 



- I fiu,y,Ol^uidC/r'/'')=Fiu,y). 

T jRd 



(94) 

(95) 
(96) 
(97) 
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This yields us the following limiting equation for S{t, y) with an arbitrary generator of 
an increasing Feller process Xt, namely A, and a generator of a stable Feller process It, 
namely L^, as r — )• 0: 



= Al 



(T-t>0) 



S{T,y)+AS{t,y) 



+ sup 



LlS{t,y) + QuMiT -t)Al^T~t>^) 



(98) 



where c(t) = tl(T-t>o)- 

In case when inner motion also takes place, we have: 



= Al 



(T-t>0) 



S{T,y)+AS{t,y) 



+ sup 



LlS{t, y) + B^S{t, y) + <7«,t(y)(T - t)Al^T-t>o) 
+5„,f(y)(Ac)(i) +F(-u,y)Al(r_t>o)] , 



(99) 



where c{t) = il(T-t>o)) for i > 0. Note that the key difference is that there are no /3's 
involved at all in this case, and the generator A is an arbitrary one describing an increasing 
Feller process Xf 

8.3 Both time and position dependent HJB 

We could have position dependence together with general time dependence: 



^n - ^n-1 + lli^n-l)^ 



^n — ^n-1 + in\^n-l) 



(100) 



(101) 



We build families of ^J and ^J as before. Also, we include waiting time rewards into 
the system. Again we use theorems 19.25 and 19.28 in [3] which guarantee that such 
approximations of the Feller processes by discrete time Markov chains are possible. Our 
boundary condition is S{T, y) and we write the DP equation: 



S'^{t,y) = sup 



[S^{T,y)+g^4y){T-t)] I ut^dr) 

T-t 



+ / / [S^t + r,y + ^)+ gu,t{yy + fiv + <?, u)] liu,r,y{dS)vt^r{dr) 

Jo JRd 

For the DP equation not to blow up, we assume that 

1 /■* 
lim — / ruT-(dr) = Ki(t), 
^^0 -r Jo 



and 



1 /"OO 

lim — 

T-s>o r 



l^ridr) = K2{t), 



(102) 

(103) 

(104) 
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and that 



fiu,y,OtJ'u,T{dC) = F{u,y). 



(105) 



This yields us the fohowing hmiting equation with arbitrary generators of Feller processes 
Xt and Yt, namely A and L, as r ^^ 0: 



+ sup 

ueu 



= A^T-t>o)S{T,y) + AS{t,y) 

LS{t, y) + guAy)iT - t)Al(r-t>o) 

+5n,t(2/)(^c)(t) +F(n,y)Al(r_t>0)] , 



(106) 



where c{t) = tl(T-t>o)- 

In case when inner motion also takes place, we have: 



+ sup 

ueu 



= Al^T-t>o)S{T,y) + AS{t,y) 
LS{t, y) + B^S{t, y) + QuMi^ - t)Alt^T-t>^) 

+(7„,t(y)(^c)(t) + F(u,y)yll(r_i>o)] , 



(107) 



where c{t) = il(T-t>o)j fo'^ i > 0. Note that the key difference is that now there are no a's 
and /3's involved at all in the equation for S{t, y), i.e. the generators of the increasing Feller 
process Xt and a Feller process Yt are arbitrary. 

9 Further research 

In our further research is proving uniqueness and existence of classical solutions to fractional 
HJB type equations with Caputo fractional derivatives, and also proving convergence of 
S'^{t,y) to S{t,y) as r — > which corresponds precisely to the limiting process. Further 
work includes direct applications of this theory to finances [TO] , [11] . 
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